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ABSTRACT 

Among the repertoire of clinical skills necessary for 
the professional development of medical students is the ability to 
create a positive doctor-patient relationship through effective 
communication skills. The purpose of this study was to create an 
instrument that reliably measures the relationship between physician 
and patient. The Relationship Instrument provides evidence to 
document both medical student ! s task and s ocio~emot i ona 1 behaviors as 
well as the socio-emotional behaviors of the patients whom they 
interview. Specifically measured were indicators of this relationship 
between third year medical students taking a clinical performance 
examination and standardized patients. Subjects (n=370) included all 
third year medical students at four medical schools in North 
Carolina. The Re ' :<t i onsh ip Instrument is a 15-item checklist that was 
completed by a standardized patient in each of 20 cases. Exploratory 
factor a <alysis did identify a two factor model that used 13 of the 
original 15 items and indicated that scores derived from the 
Relationship Instrument correspond to the two dimensions used to 
construct it. Factor scores on the instrument were reliable. Thus, 
evidence suggests that measurements derived from the Relationship 
Instrument are consistent with interpretation of the two components 
of the doctor-pat ient relat i onship. A copy of the instrument is 
included. (JBJ) 
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Relationship Skills in a Clinical Performance Examination: 
Reliability and Validity of the Relationship Instrument 

Cynthia Bolton, Donna Harward, Allen Smith, 
Gwendie Camp, Emil Petrusa, Boyd Richards, Steve Willis 



Introduction 

Among the repertoire of clinical skills necessary for the professional 
development of medical students preparing to become physicians is the ability to 
create a positive doctor-patient relationship through effective communication and 
interpersonal skills. It has been argued that the skills which facilitate the doctor- 
patient relationship build the necessary bonds of understanding which both the 
physician and patient require in order to work toward greater health of the patient 
(Korsch & Negrete, 1972; Davis, 1971; Korsch, et al, 1968). Furthermore, a 
productive and satisfying doctor-patient relationship results in improved 
communication, an increase in patient compliance, attentiveness, and in an 
improved patient perceived professional demeanor (Barrows, 1992; Hall, et al, 
1988a). Although some communication skills can be thought of as case specific, 
because different communication skills may be required for different patients in 
different contexts, an argument has been made for the existence of generic 
relationship skills which can promote patients' understanding of and eventually 
their participation in behaviors that will improve their health status (Bolton, 1994). 
The Relationship Instrument was developed to provide evidence to document both 
the medical student s task and socio-emotional behaviors as well as the socio- 
emotional behaviors of the patients whom they interview. 
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Design 

The purpose of this study was to create an instrument that reliably measures 
the relationship between physician and patient. The Relationship Instrument was 
designed specifically to measure indicators of this relationship between third year 
medical students taking a clinical performance examination (CPX) and standardized 
patients (SPs). Confidentiality was maintained by assigning students a test number 
in which scores on the measures were recorded. 

The data were collected during the summer and early fall of 1993. The CPX is 
a two-day twenty-station SP-based standardized examination of clinical skills. The 
exam assesses the "clinical skills" of the students using medical simulations. 
Research studies have reported performance based examinations of clinical skills 
with standardized patients as a reliable and valid method of assessing the clinical 
skills of medical students (Hardin, et al., 1975, Van der Vleuten & Swanson, 1990; 
Vu, et al, 1992). Because this study was nested in a larger research project 
investigating the development and implementation of the CPX, permission to 
conduct the study was granted by each medical school, the NCMSC, and test 
administrators. 

The design of this study was developed to investigate the following research 
hypotheses: 

1. An exploratory factor analysis will indicate that the scores derived from the 
Relationship Instrument reflect a factor structure that corresponds to the two 
domains used to construct the instrument. 

2. An internal consistency reliability analysis using Cronbach's alpha will 
indicate that the reliability of each factor measured by the Instrument will be greater 
than .70. 
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Subjects and Setting 

This study analyzed data collected during the initial implementation of the 
CPX at the four medical schools in North Carolina: Bowman Gray School of 
Medicine, Duke University, East Carolina University, and The University of North 
Carolina at Chapel Hill, which comprise the North Carolina Medical Schools 
Consortium (NCMSC). The NCMSC was created to develop, implement and 
evaluate innovative educational projects for teaching and evaluating medical 
students. 

The subjects for this study included all third year medical students at the four 
medical schools who had completed the majority of their third year, and their 
clinical clerkships including Internal Medicine, Surgery, Obstetrics and Gynecology, 
and Pediatrics (with the exception of Site 3, where students finish clerkships after 
the second year). 

The data for this study was based on 370 third year medical students from 
four medical schools provided by the NCMSC. The total number of students at the 
four sites was 440, but 70 were absent or excused because of unavailability (i.e. out of 
the state or country), illness, or incompletion of third year clerkships (i.e. part-time 
students). The available sample consisted of 60% males and 40% females. Eighty 
point five percent of the students were Caucasian, 9% were Asian, 8.6% were 
African American, 3% were Native American, and 3% were Hispanic or ''Other". 
Three out of the four schools participating provided birth year information. Thirty- 
seven percent of the students for whom age data were available were between the 
ages of 24-26, and 36.5% were 27 years of age or older. The birth year data were 
missing for 26.5% of the students. (See Table 1). 
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Table 1: Sample Frequency Distribution by Gender, Race, and Age 







Frequency 


Percent 


Gender 


Male 


221 


59.7 




Female 


149 


An o 
4U.3 




Total 


370 


100 


Race 


Caucasian 


298 


80.5 




African American 


32 


8.6 




Asian 


34 


9.2 




Other 


6 


1.7 




Total 


370 


100 


Age 


24-26 years old 


137 


50.4 




27 and older 


135 


36.5 




Missing 


98 


13.1 




Total 


370 


100 



Instrumentation 

The Relationship Instrument is a 15 item checklist that was completed 
by the corresponding SP in each of the twenty cases. The SP had limited time 
(sometimes less than 5 minutes) in which to complete the Relationship Instrument 
along with other CPX instruments, therefore minimizing the number of items thai 
could be included. Since SPs were recruited from the general public at large, and not 
necessarily from a medical population, the language used on the instrument was 
designed to be easily understood and usable by a large population of SPs with 
minimum training. The instrument requires SPs to read each item and respond on 
a four point rating scale how much the SP agrees or disagrees with the specific item 
descriptions of the encounter. For the purpose of this study the following scoring 
schema was used: 1= Strongly Agree; 2 = Agree; 3 = Disagree; 4= Strongly Disagree. 

The Relationship Instrument was developed by a panel of educational and 
medical experts who wrote the items for the instrument based on clinical and 
assessment experiences and a review of the literature on doctor-patient 
relationships. The construction of the Relationship Instrument was theoretically 
based on research conducted by Hall, et al, (1988b). In this meta-analysis of doctor- 
patient interactions, physician behaviors were separated into two domains: task and 
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socio-emotional behaviors. The ability to measure behaviors which effect patient 
behavior is a strong step forward in medical education, particularly clinical 
assessment, since studies have shown correlations between physician behaviors and 
patient task behaviors such as compliance and recall (Hall, et al, 1987; Bartlett, et al, 
1977), and socio-emotional behaviors such as satisfaction (Hall, et al, 1988a; Shortell, 
et al, 1977). 

After reviewing the literature, 28 behavioral indicators of 
interpersonal /relationship skills and 37 communication/patient education 
indicators were listed. From this list, 24 items (14 relationship, 10 communication) 
were selected and then reviewed by the panel and reduced again to 22 (10 
relationship, 12 communication). After items were further revised, the 
Relationship Instrument was reviewed by the Test Committee who revised some 
items and reduced the final number to 15: 8 Socio-Emotional items and 7 
communication items. An example of a Socio-Emotional item is Item 1 "The 
student physician appeared empathetic (seemed sensitive, understood my problems, 
etc.)". An example of an item representing communication task behaviors is Item 
10, "The student physician summarized what I said." (See Appendix for complete 
Relationship Instrument). 

The data were analyzed to provide evidence for the reliability and validity of 
the scores obtained from the Relaitonship Instrument. The analyses completed for 
this study included factor analysis and reliability estimates. 

Item Screening 

An initial screening of the item scores indicated that the assumptions of 
normality were within satisfactory limits. Item means, standard deviations, 
skewness and kurtosis are listed in Table 2. 
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Table 2: Descriptive Statistics for Relationship Instrument Items Used in Analyses 



Item 


Mean 


Standard 
Deviation 


Skewness 


Kurtosis 


1 


1.588 


.678 


.950 


.612 


J 




.586 


.866 


.769 


-x 

J 


1 549 


.641 


.901 


.442 


A 


1 674 

JL . \J / *± 


.689 


.721 


.104 




1.656 


.678 


.738 


.174 


6 


1.640 


.671 


.766 


.273 


7 


1.640 


.653 


.753 


.487 


8 


1.777 


.782 


.683 


-.257 


9 


1.749 


.693 


.544 


-.190 


10 


1.923 


.783 


.454 


-.419 


11 


1.603 


.645 


.855 


.798 


12 


1.705 


.703 


.695 


.058 


13 


1.990 


.791 


.362 


-.515 


14 


1.965 


.813 


.437 


-.500 


15 


1.994 


.804 


.284 


-.763 


Average 


25.87 


8.21 







Total Score 



*Item Score Range: l=Strongly Agree, 2=Agree / 3=Disagree, 4=Strongly Disagree 



Exploratory Factor Analysis 

Exploratory factor analytic procedures utilizing principal axis method with an 
oblique rotation were used to determine the number and nature of the underlying 
constructs of these items. The correlational matrix for the principal components 
analysis revealed no variables with correlations of <.3 indicating that there was at 
least 10% shared variance among factors and warranted an oblique rotation 
(Tabachnick & Fidell, 1989). Another test of the factorability of the data was 
provided by the Kaiser-Meyer-Olken (KMO) measure of sampling adequacy. A 
value of .6 or higher is required for a good factor analysis (Kaiser, 1974). The KMO 
for this data was .965, which indicated that this sample was satisfactory for common 
factor analysis. Furthermore, an examination of the residuals in the reproduced 
correlation matrix revealed that the residuals were small and did in fact adequately 
explain the data since small residuals indicate a close fit between observed and 
reproduced matrices (Table 3). 
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Eigen values derived from the principal components analysis produced two 
factors with eigen values exceeding one (eigen values less than one are usually not 
important to the variance of the factor). A two factor solution and a three factor 
solution were used to rotate the items using an oblique rotation procedure. 

The two factor solution was the clearest and most parsimonious. The results 
of the oblique rotation revealed a clear picture of the nature of the variables (only 
one item did not load on either factor). Using a cut off of .45 for inclusion of a 
variable on a factor since "[t]he greater the loading, the more the variable is a pure 
measure of the factor" (Tabachnick & Fidell, 1989), the results indicated that all the 
items loading on Factor 1 were the same items that were originally proposed to 
measure Socio-Emotional Behaviors. The highest loading item on Factor 1 was 
Item 7: "As a patient, I felt safe in expressing myself and describing my situation." 
The items that loaded most strongly on the Socio-Emotional factor were those 
designed to reflect the interpersonal skills which take into account the patient's 
social and emotional well being. 

The items loading on Factor 2 were the same items that were originally 
proposed to measure Communication Task Behaviors. These behaviors are 
intended to clarify and educate specific concerns to the patient. For example, Item 13 
which loaded the highest on Factor 2 states: "The student physician checked my 
understanding of what s/he said." 

An examination of the rotated pattern revealed two factors that were 
moderately correlated (.764) (see Figure 1 for a graph of the factor analysis with item 
coordinates). There was one borderline item (Item 9) that loaded .489 on Factor 1 
and .356 on Factor 2, and one item (Item 11) which failed to load on either factor 
(loaded .387 on Factor 1 and .349 on Factor 2). Further examination of these two 
items revealed that these two items were double worded items, and included the 
conjunction "and". Therefore, these two items were dropped for all remaining 
analysis. Table 4 reports the factor structure of the Relationship Instrument and the 
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loading of each item as well as the eigen values. Table 5 lists the original items and 
the revised classification of items on the instrument based on the exploratory factor 
analysis. Thus, the two factors identified in the exploratory factor analysis seem to 
match the constructs that were originally proposed. 




t 



Item Coordinates Item Coordinates 

1 .79224 .28967 9 .59657 .55140 

2 .76832 .24028 10 .32651 .71486 

3 .78691 .32775 11 .49417 .52540 

4 .79145 .33137 12 .49695 .60564 

5 .70340 .43546 13 .29857 .71486 

6 .75587 .38148 14 .22978 .82138 

7 .79145 .32906 15 34680 .70678 

8 .71278 .444154 



Figure 1: Factor Analysis Graph of Item Coordinates 
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Table 4: Relationship Instrument Item Factor Loadings 



Extracted Rotated 



Item 


Factor 1 


Factor ? 




Farfnr 7 


i 
i 


7Q7 


- 97A 


fm7 




Z 


7/11 


".Z77 




-.uoo 




ftl 7 
.01/ 


~.Z*i*i 


RAA 


-.U lo 


A 


.OZJ} 


-.Z*i*l 


•sw 


-.uzz 


D 


• OZU 


-. 1U/ 




1 7n 
. l/U 


6 


827 




780 




7 


. OZ. 1 






~.uzo 


8 


.833 


-.106 


.685 


.173 


9 


.810 


.049 


.489 


.356 


10 


.705 


.347 


.120 


■633 


11 


.715 


.094 


.387 


.349 


12 


.768 


.155 


.342 


.465 


13 


.743 


.438 


-.036 


.859 


14 


.697 


.491 


-.086 


.858 


15 


.716 


.238 


.152 


.611 


Percent of 


60.5 


7.3 


58.0 


4.9 



Variance 



Eigen Values 9.068 1.098 



Table 5: Proposed and Revised Classification of Relationship Instrument Items 

Proposed Classification of Items on the Relationship Instrument 
Category Items 
Socio-Emotional Behaviors 1, 2, 3, 4, 5, 6, 7, 8 

Communication Task Behaviors 9, 10, 11, 12, 13, 14, 15 

Revised Classification of Items on Relationship Instrument Based on Exploratory Factor Analysis * 
Category Items 
Socio-Emotional Behaviors 1, 2, 3, 4, 5, 6, 7, 8 

Communication Task Behaviors 10, 12, 13 , 14, 15 

* Includes Only Items Loading .45 or Higher for the 2 Factor Solution 
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Internal Consistency Reliability Estimates 

Three different internal consistency reliability estimates were calculated in 
this investigation since the Relationship Instrument can be used and interpreted in 
a number of different ways in the context of performance assessment. Cronbach's 
coefficient alpha was used in each analysis. For the first design, the internal 
consistency reliability estimates are computed for the scores which are based on the 
ratings each student received on each of the two factors of the Relationship 
Instrument in twenty different medical contexts (cases) with a different trained 
observer for each case (n=7821). Students who were not rated (i.e. missing values) 
on one or more items for that particular dimension were deleted from the analysis 
(Factor 1: n = 6702; Factor 2: n = 6758). The scores were totaled on each factor by 
adding the ratings across item occurrences. The internal consistency reliability 
estimates for this model were .94 for the Socio-Emotional Behavior factor and .87 for 
the Communication Task Behaviors factor (Table 6). 

The second design averaged item scores for each factor for each student across 
the twenty different medical contexts (cases) and then computed an alpha coefficient 
for these two averaged factor scores (n = 370). Averaged factor scores were obtained 
by computing the mean factor score for each student. Using this model the internal 
consistency reliability estimate for Socio-Emotional Behaviors was .98; 
Communication Task Behaviors, .94 (Table 7). 

The third design calculated separate reliability estimates for each item across 
the twenty medical cases. This statistic was computed since it may be of interest to 
know how consistently each item operates across very diverse medical contexts 
(cases) during the performance examination. In other words, this analyses 
computed how stable each item was given the diverse nature of the cases (e.g. 
emergency room, doctor's office, hospital room) with different patient attributes (e.g. 
age, race, gender, socio-economic status) presenting with a variety of complaints. 
Using this model, item coefficient alpha estimates ranged from .51 to .73 (Table 8). 
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Table 6: Internal Consistency Reliability Estim ates Design 1: All Occurrences 

Relationship Instrument Using All 15 Items 

Scale # of Items Reliability 

Total 15 .95 

Revised In strument 

Scale # of Items RjdLaMiiy 

Socio-Emotional Behaviors 8 .94 

Communication Task Behaviors 5 .87 



Table 7: Internal Consistency Reliability Estimates Design 2: Average Occurrences 

Scale # of Items Reliability 

Socio-Emotional Behaviors 8 .98 

Communication Task Behaviors 5 .94 



Table 8: Internal Consistency Reliability Estimates Design 3: Item Estimates Across 
Cases 

Scale Item Reliability 

Socio-Emotional Behaviors 1 .73 

2 .51 

3 .62 

4 .67 

5 .66 

6 .59 

7 .60 
8 71 

Communication Task 

Behaviors 10 .60 

12 .66 

13 .60 

14 .61 

15 .64 
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Discussion 

Instrumentation Structure 

Exploratory factor analysis did identify a two factor model that used 13 of the 
original 15 items. An oblique rotation was used because it was believed that the 
factors defining the construct would be correlated. Analysis revealed that there was 
a moderate correlation of .74 between the two factors. Factor 1 was described as the 
Socio-Emotional Behaviors Scale. The items on this factor were most reflective of 
those skills which take into account the patient's social and emotional well being, as 
well as the resulting feelings of the patient due to the physician's skill. For example, 
Item 4 which loaded highly on factor 1 states "As a patient, I felt that the student 
physician understood me". Factor 2 was described as Communication Task 
Behaviors Scale. These items reflect the behaviors that facilitate the transmission of 
knowledge by engaging the patient in the relationship process. Item 14 which 
loaded highly, on this factor states "The student physician helped me take an active 
role in my care". 

Using a standard to retain an item that it must load on a factor > .45, one item 
failed to load on either factor, and another loaded on both. Both of these items 
contained the conjunction 'and' making them double worded items. These items 
were dropped from further analysis and should be dropped from the instrument or 
revised for future uses of the Clinical Performance Examination. These items could 
be revised into two separate items. Given the clarity of the factor loadings, it was 
concluded that the two factor structure derived from exploratory factor analysis 
could be interpreted as being closely related to what was intended when the 
instrument was constructed. The research hypothesis that an exploratory factor 
analysis will indicate that the scores derived from the Relationship Instrument 
corresponds to the two dimensions used to construct the instrument was supported. 
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Reliability 

For both factors on the Relationship Instrument, the internal consistency 
reliability was higher than the .70 specified as the minimally accepted standard for 
using scales for group studies. The eight item Socio-Emotional Behaviors Scale had 
a reliability coefficient of .94, and the five item Comunication Task Behaviors Scale 
had a reliability coefficient of .87. Two other reliability estimates were calculated 
due to different possible uses and interpretations of the scores. When the item 
scores were averaged for each student across cases, the reliability coefficients 
increased on both scales: Socio-Emotional was .98 and Communication was .94. 
These findings support the research hypothesis that the factor scores on the 
Relationship Instrument were reliable. 

Internal consistency estimates were also computed for each item on the 
instrument when used across cases. The resulting alpha coefficients were moderate 
in nature. These findings suggest that the scores for each item were stable when 
summed across cases. However, when examining the consistency of the items 
across cases, only two items exceeded the .70 minimum standard, although most of 
the remaining items were very near the minimally accepted standard. This result 
could be due to the diversity of contexts for each case, as well as patient attributes 
and presenting complaints. Items worded in more general terms did produce 
greater consistency across cases. For instance, Item 8 "As a patient, I felt that this 
encounter could be the beginning of a good doctor-patient relationship" can be 
easily generalized across cases, and produced a reliability consistency estimate of .71. 
Whereas, Item 2 "The student physician appeared respectful (used my name, used 
language I could understand without speaking down to me, etc.)" may not be as 
stable in an emergency situation in comparison to a routine examination. This 
item produced a consistency estimate of .51. Also, the two items with high 
reliabilities were ones that dealt directly with empathy and patient satisfaction (basic 
elements of socio-emotional behaviors) and therefore translate well across medical 

ERLC ' 
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contexts. The items on the Communication Task Behaviors Scale were all in the .60 
to .66 range due to the specificity of the behaviors being meausred. 

Recommendations 

The evidence suggests that the measurements derived from the Relationship 
Instrument are consistent with the interpretation of the two components of the 
doctor-patient relationship. Analysis supports the continued use of the 
Relationship Instrument to measure Socio-Emotional Behaviors and 
Communication Skills of student physicians across 20 cases in a clinical 
performance examination; however, more research and training in conjuction with 
the Relationship Instrument is needed. 

A confirmatory factor analysis study on the revised instrument should be 
used to cross validate the two factor model. Confirmatory factor analysis would 
allow researchers to test the extent to which data produced by the Relationship 
Instrument fits the hypothesized model and then to observe how closely that model 
can reproduce the observed data by comparing variance and covariance correlations. 

Items that have double meanings should be avoided since they are 
ambiguous to raters. Likewise, raters need to be carefully trained to use the total 
range of the instrument's scale, since raters tended to underutilize the upper levels 
of the scale (Disagree, Strongly Disagree). This consideration is important since it 
may be a natural response for non-experts to rate all students as 'good' in order to 
avoid the uncomfortable feelings elicited from rating a student poorly. This 
response may hinder SPs from making true judgments and result in a failure to 
discriminate between students' skills. Thus, training the raters to use the entire 
range of the scales on the instrument will result in better discrimination between 
student physician behaviors. This is a critical factor in garnering reliable and valid 
data on this instrument. 



Relationship Instrument 

17 

Finally, further research is needed that more fully investigates the underlying 
construct(s) of the doctor-patient relationship. Specifically analyses of Relationship 
Instrument scores in regards to gender, race and age will add to the 'nomonolgical 
net' which may investigate the arguments for such a construct. Furthermore, it 
might be particularly useful to continue to examine how the gender, age, and race of 
the SP rater may interact with student variables in predicting scores since this is a 
possible threat to the validity of the exam. 
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Physician Name. 



Remember to: 

Fill in ALL the blanks above, including the code bubbles. 
Complete all items on the checklist— don't skip any! 
For each item, darken one bubble per item using the given scale 
Write comments below as desired. 

Strongly Disagree 
Disagree 
Agree 
Strongly Agree 

THE STUDENT PHYSICIAN APPEARED... 

empathetic (seemed sensitive, understood my problems, etc.) 

respectful (used my name, used language I could understand without speaking down to me, 

etc.) 

interested (listened to my story, followed up on my comments and questions, etc.) 

AS A PATIENT, I FELT... 

that the student physician understood me 

that I would respect the advice the student physician gave me 

comfortable asking for additional information or clarification 

safe in expressing myself and describing my situation 

that this encounter could be the beginning of a good doctor-patient 
relationship 

THE STUDENT PHYSICIAN... 

helped me explain my situation and clarify my concerns 

summarized what I said 

gave me opportunities to ask questions and make comments 
provided clear information 
checked my understanding of what s/he said 
helped me take an active role in my care 
asked about my needs and interests 
WRITTEN COMMENTS: 

9.3 
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